Image processing method and image capturing apparatus

ABSTRACT

An image processing method includes acquiring a plurality of image data items by continuously capturing images of an object by an image capturing apparatus (S1), transmitting the plurality of image data items to a computing section (S2), computing image feature values each of which relates to positional information and information correlated with illumination conditions of each of the image data items by using the computing section (S3), comparing degrees of matching between reference image feature values, each of which corresponds to positional information and information correlated with illumination conditions, and the plurality of image data items (S4), and selecting, from the plurality of image data items, an image having positional information and illumination conditions that match the reference image feature values and outputting the image (S5).

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.16/296,051, filed Mar. 7, 2019, which is a Continuation of U.S. patentapplication Ser. No. 14/696,213, filed Apr. 24, 2015, now U.S. Pat. No.10,311,592, which claims the benefit of Japanese Patent Application No.2014-093086, filed Apr. 28, 2014, all of which are hereby incorporatedby reference herein in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a method and an apparatus for capturingan image of an object by using a camera. More particularly, the presentinvention relates to a technology for capturing an image of an objectthat moves with respect to a camera.

Description of the Related Art

In an automatic assembly line for a product using a robot, an operationin which the robot holds a component, which has been supplied, andmounts the component, which has been held by the robot, on a workpieceis repeated. In the case where the supplied component is not strictlypositioned, measurement of the position and orientation of the componentby using a camera may sometimes be performed in a process for holdingthe component.

This measurement is performed in order to bring the position andorientation of the component, which has been held by the robot, relativeto the robot as close to a target position and a target orientation aspossible. In other words, “a predetermined way in which the robot holdsa component”, which is a target value, and “the actual way in which therobot holds a component”, which is an actually measured value, arecompared with each other, and in the case where there is a differencebetween the target value and the actually measured value, the robotperforms a process for causing the actually measured value to be thesame as the target value. This process is performed in order to achievean improvement in the accuracy with which assembly is performed by therobot in a process that is to be performed after the robot has held acomponent.

An exemplary specific method is as follows: the robot holds a component,then transports the component to a position in front of a camera, andafter that, stops its operation. A still image of the component iscaptured by the camera, and the captured still image undergoes imageprocessing, so that the position and orientation of the componentrelative to the robot are recognized. The robot performs a process forchanging its orientation and a process for changing the way in which thecomponent is held on the basis of the recognition results, so that theposition and orientation of the component are changed to be within arange in which a process for mounting the component on a workpiece canbe performed. After that, an assembly operation is performed.

In recent years, there has been a need to automatically assemble aproduct by using a robot with good efficiency while using visualinformation obtained by a camera. Thus, in order to achieve a furtherimprovement in productivity per hour, it is effective to reduce as muchas possible an increase in the time taken for transportation of acomponent due to slowing down, interruption, actuation, and speeding upof a robot, which occur during image capturing. Ultimately, it isdesirable that an image of the component be captured while the componentis moving in front of the camera without stopping the robot fromtransporting the component (this method will be hereinafter referred toas a method for capturing an image of a moving object).

In the method for capturing an image of a moving object, a sample imageof an object (a target value) that has been previously stored and animage of the object captured while the object is moving (an actuallymeasured value) are compared with each other, and the object to bemeasured is recognized in accordance with the level of similaritybetween the values.

In order to perform an image capturing of a moving object with highrecognition accuracy, an image processing algorithm is determined, andin actual measurement, an image of an object needs to be captured whilethe object is moving at a resolution similar to that in a sample imageof the object that has been stored as a target value in such a manner asto correspond to the image processing algorithm.

In addition, an image of the object needs to be captured underillumination conditions, such as reflections of shadows and ambientlight, that are as similar to the illumination conditions under whichthe sample image has been acquired as possible.

For example, assume the case where an image processing algorithm thatmeasures the position and orientation of a component, which is anobject, by utilizing a mark placed on the component is used. In thiscase, in actual image capturing, the light of fluorescent lampsinstalled inside a factory may sometimes be reflected off a surface ofthe component, and glare may sometimes be produced. In addition, animage capturing may sometimes be performed under illumination conditionsunder which the mark is invisible to a camera due to the influence ofsuch glare, and there is a possibility that the position and orientationof the component will be falsely recognized.

In the field of such image capturing of a moving object, as described inJapanese Patent Laid-Open No. 9-288060 (hereinafter referred to asPatent Document 1), a technology for extracting an image suitable forimage analysis by capturing images of a moving component has beendesigned.

The following technology is described in Patent Document 1. In thetechnology described in Patent Document 1, a plurality of images of anobject that is conveyed in front of a camera by a conveyor arecontinuously (or intermittently) captured in accordance withimage-capturing-trigger signals sent from a control unit of the cameraat regular intervals. After that, an image in which the distance betweenthe object and the center of the image is the smallest is selected asthe best image among the plurality of captured images, and then, thebest image is output.

Operation of a camera described in Patent Document 1 is as follows:first, the last captured image and an image captured before the lastcaptured image are stored in a memory of the camera. In this state, oneof the following is selected on the basis of the displacement amount ofthe position of an object with respect to the center of the image, whichis calculated in each of the two images: (1) the last captured image isdetermined as the best image, (2) the image captured before the lastcaptured image is determined as the best image, and (3) thedetermination process is suspended, and the image capturing operationcontinues. In the case where the displacement amount falls below apredetermined threshold, that is, in the case where the object issufficiently near the center of the image, (1) is selected. In the casewhere the displacement amount in the last captured image is larger thanthe displacement amount in the image captured before the last capturedimage, that is, in the case where the object, which has come near thecenter of the image once, has passed through the center of the image andis moving in a direction away from the center of the image, (2) isselected. In the case where neither of the above cases holds true, (3)is selected. As a result, in the case where capturing an image of amoving object is performed by using the technology of the related artdescribed in Patent Document 1, an image in which the distance betweenthe position of a component and a position predetermined by a user(hereinafter sometimes referred to as target capturing position) is thesmallest can be retrieved from images captured by a camera.

However, in the method for capturing an image of a moving objectdescribed in Patent Document 1, an image to be output is selected amongcaptured images while only the position of a component functions as areference. Thus, in the case where an image of a component whoseorientation has been inclined due to vibration and the like generatedduring transportation of the component is captured, the image of thecomponent is captured under conditions under which the position andorientation relationship between the component and thecamera/illumination is different from the position and orientationrelationship when the image processing algorithm is determined.Therefore, an image captured under illumination conditions that aredifferent from those originally assumed is acquired, and there is apossibility that the brightness in a portion of the acquired image maybe different from the brightness that has been assumed, and that contourinformation of an area different from the area that has been assumed maybe strongly detected. In such a case, in the case where a process forimage processing and a process for recognition of an object areperformed, there is a possibility that the position and orientation ofthe component will be falsely recognized, which in turn leads to failureof automatic assembly.

SUMMARY OF THE INVENTION

The present invention has been made in view of the above problems.According to an aspect of the present invention, a method is providedfor capturing an image of a moving component under illuminationconditions similar to those under which an image with which an imageprocessing algorithm is determined beforehand is acquired.

According to another aspect of the present invention, an imageprocessing method includes acquiring a plurality of image data items bycontinuously capturing images of an object by an image capturingapparatus, transmitting the plurality of image data items to a computingsection, computing image feature values each of which relates topositional information and information correlated with illuminationconditions of a respective one of the image data items by using thecomputing section, and outputting an image selected from the pluralityof image data items. In the outputting an image, degrees of matchingbetween the positional information and a reference image feature valuethat is stored in a memory are compared, degrees of matching between theimage feature values, which have been computed, and a reference imagefeature value that is stored in the memory and that relates toinformation correlated with illumination conditions are compared, and animage having positional information and illumination conditions thatmatch the reference image feature values is selected from the pluralityof image data items and output.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an apparatus for capturing animage of a moving object according to a first embodiment of the presentinvention.

FIG. 2 is a flowchart according to the first embodiment of the presentinvention.

FIG. 3 is a diagram illustrating an example of an image data itemaccording to the first embodiment and a second embodiment of the presentinvention.

FIG. 4 is a diagram illustrating examples of captured images accordingto the first embodiment of the present invention.

FIG. 5 is a diagram illustrating examples of binarized images accordingto the first embodiment of the present invention.

FIG. 6 is a block diagram illustrating an apparatus for capturing animage of a moving object according to the second embodiment of thepresent invention.

FIG. 7 is a diagram illustrating examples of captured images accordingto the second embodiment of the present invention.

FIG. 8 is a diagram illustrating examples of inter-frame differences ofimages that have been converted into binary representations according tothe second embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS First Embodiment

An apparatus for capturing an image of a moving object according to afirst embodiment of the present invention will be described. In thefollowing description, a process of an image processing method accordingto the present invention will be described step by step, and after that,individual components included in a camera and the like will bespecifically described with reference to an exemplary embodimentcorresponding to the first embodiment.

FIG. 1 illustrates the configuration of the apparatus for capturing animage (of a moving object), which will be sometimes referred to as imagecapturing apparatus, according to the first embodiment of the presentinvention. As illustrated in FIG. 1, the apparatus for capturing animage (of a moving object) includes a camera 114, which includes animage pickup element 101, a lens 102, and a computing section 115, and arobot hand 113 that is used for holding a component 112. The computingsection 115 includes a controller 103, an image input interface 104, animage-feature-value-calculation unit 106, a memory 107, an imageselection unit 108, and an image output interface 109.

First, the flow of processes when the image capturing apparatus capturesan image of the component 112, which is moving while being held by therobot hand 113, by using the camera 114 will be briefly described, andthen, signals in each component and the operation of each component willbe described in detail.

Initial setting to be performed before performing capturing of an imageof a moving object will be described first. The controller 103 sets amovie recording mode in which the image pickup element 101 continuously(or intermittently) captures images as an operation mode of the imagepickup element 101 through communication using a serial peripheralinterface (SPI) or the like. The following two predetermined imagefeature values based on experiments, designed values, or the like arestored in a storage area of the controller 103. In other words, an imagefeature value, which is a target capturing position, and an imagefeature value that is correlated with illumination conditions at thetarget capturing position are stored as reference image feature values,and these reference image feature values are output beforehand to theimage selection unit 108. Regarding an area in which data items such asthe target capturing position and the image feature values are to bestored, a particular portion of the computing section 115 that has astorage function may be arbitrarily set as a “memory”.

Note that details of a method for calculating the image feature valuecorrelated with the illumination conditions will be described later. Therobot hand 113 is instructed beforehand to move in such a manner thatthe component 112 crosses the visual field range of the camera 114 whilethe component 112 is being held by the robot hand 113. In the memory 107in which an image data item and an image feature value are to betemporarily stored, an image having only pixels whose luminance valuesare “0” and an image feature value of “0” are stored as initial dataitems. The initial setting in the present embodiment is completed byperforming these operations. Note that the target capturing position andthe reference image feature values need not be stored in the storagearea of the controller 103 and may be stored in the memory 107 andoutput to the image selection unit 108.

The flow of processes of acquiring image data items and image processingwhen capturing of an image of a moving object is performed will now bedescribed with reference to the flowchart illustrated in FIG. 2.

(S1 Acquisition of Plural Image Data Items)

The operation mode of the image pickup element 101 has been set to themovie recording mode by the initial setting, and the image pickupelement 101 acquires a plurality of image data items by continuously(intermittently) capturing images of the component 112 through the lens106 until the component 112 reaches the vicinity of the target capturingposition.

The image pickup element 101 is not limited to having the movierecording mode as its operation mode and may acquire a plurality ofimage data items by continually capturing sequential images of thecomponent 112 at certain intervals. In the following description, theterm “continuously capture images” also refers to image capturingperformed in this manner.

(S2 Transmission of Image Data Item to Computing Section)

The captured images are sequentially output to the image input interface104 as, for example, LVDS signals. In this case, the LVDS signals aretransmitted through, for example, 10 pairs of differential signal lines,and the differential signal lines output serial signals, which are theLVDS signals multiplied by seven and serialized. The image inputinterface 104 sequentially converts image data items that arecontinuously input to the image input interface 104 as the LVDS signalsinto image data items in the form of parallel signals and outputs theimage data items to the image-feature-value-calculation unit 106 of thecomputing section 115.

(S3 Compute Image Feature Values of Illumination Conditions andPositional Information)

The image-feature-value-calculation unit 106 calculates image featurevalues of the image data items, which are input to theimage-feature-value-calculation unit 106. In the present embodiment, twotypes of image feature values are used. One of the image feature valuesis an image feature value that relates to information that is correlatedwith positional information, which indicates the position of thecomponent 112 in a captured image. The other one of the image featurevalues is an image feature value that is correlated with illuminationconditions, which indicate an illumination environment in which imagecapturing is performed.

Here, illumination conditions denote, for example, the position andorientation of the component 112 with respect to the camera 114 andambient light sources (not illustrated) such as lighting for use inimage capturing (not illustrated) and a fluorescent lamp. As a specificexample, there is a case where an inclination of the component 112 withrespect to the camera 114 is larger than the inclination of thecomponent 112 with respect to the camera 114 that has been assumed byusing the image processing algorithm, so that illumination conditionsare different from predetermined illumination conditions. In such acase, image brightness may sometimes be affected as a result of changesin the intensity of ambient light of the fluorescent lamp and the likereflected in an image, a position at which such ambient light isreflected, and the intensity of scattered light that is captured. Imagebrightness may also sometimes be affected as a result of changes insharpnesses of a feature point and a contour line of a marker for use inimage processing or the like, shading, and a position at which a shadowis generated.

The image feature value correlated with illumination conditions may bedetermined in the following manner.

First, images of the component 112 are acquired by capturing while thecomponent 112 is inclined or the like in such a manner that theorientation of the component 112 with respect to the camera 114 differsbetween the images. In addition, it is confirmed whether there is animage feature value that significantly changes in accordance with theorientation of the component 112 among various image feature valuesincluded in data items of the images. The image feature value correlatedwith illumination conditions can be determined beforehand through theseprocesses by a user who performs setting.

Examples of the image feature value correlated with illuminationconditions include a total value of the luminance of an image, a totalvalue of the luminance in image areas obtained by dividing an image intoa plurality of blocks, the coordinate value of a feature point, thedensity gradient value in a particular image area, and the intensityvalue of a contour line. It is most desirable to use an image featurevalue that is used when the position and orientation of the component112 are recognized by using images that have been captured by the imagecapturing apparatus. Alternatively, a plurality of image feature valuesmay be used by, for example, normalizing the plurality of image featurevalues and calculating the average of the plurality of image featurevalues. In the following description of the present embodiment, as anexample, a total value of the luminance of an image is used as the imagefeature value correlated with illumination conditions.

The image-feature-value-calculation unit 106 of the computing section115 calculates these image feature values and outputs these imagefeature values with a corresponding image data item to the imageselection unit 108. Details of a method for calculating the position ofthe component 112 and a method for calculating a total value of theluminance of an image will be described later. In the case where theposition of the component 112 is not detected in an image, that is, inthe case where the component 112 is not present in the field of view ofthe camera 114, the image-feature-value-calculation unit 106 outputs “0”as positional information to the image selection unit 108.

(S4 Matching Between Reference Image Feature Value and ReferencePositional Information)

While the image data item and the corresponding two image feature valuesare input to the image selection unit 108 from theimage-feature-value-calculation unit 106, image feature values that arerespectively correlated with positional information and illuminationconditions are input to the image selection unit 108 from the controller103. The image selection unit 108 refers to the image feature values,which have been input thereto, and in the case where the component 112has not yet entered in the field of view of the camera 114, that is, inthe case where the positional information of the component 112 is “0”,the image selection unit 108 does not perform an operation and waits forthe next image data item and image feature values. Once the component112 has entered in the field of view of the camera 114, and thepositional information of the component 112 has been changed to a valueother than “0”, the image selection unit 108 compares the referenceimage feature values stored in the memory 107 and the image featurevalues that are extracted from the image data item of the capturedimage. Then, the image selection unit 108 performs matching processingfor determining whether given determination conditions are satisfied.

Then, among the image data items of the captured images, an image dataitem from which image feature values closer to the reference imagefeature values are extracted is selected. In the case where the imagefeature values that have been input are closer to the reference imagefeature values than the image feature values stored in the memory 107,the contents of the memory 107 are updated to the image data item andthe image feature values that have been input. In addition, when thecomponent 112 is moved outside the field of view of the camera 114, andthe positional information of the component 112 is changed to “0” from avalue other than “0”, an image data item stored in the memory 107 at thetime is read and output to the image output interface 109.

(S5 Output of Image)

The image data item, which has been input to the image output interface109 and which is in the form of a parallel signal, is multiplied by, forexample, seven and serialized by the image output interface 109, andthen the image data item is output as an image signal through the 10pairs of differential signal lines in accordance with an image signalstandard such as Camera Link. The image signal, which has been output,is received and processed by an external frame grabber board or thelike.

Through the above processes, when the component 112 passes through thetarget capturing position, an image data item having image featurevalues that are the closest to a sample image can be captured, and thus,the probability of failure to recognize the position and orientation ofthe component 112 can be reduced.

EXEMPLARY EMBODIMENT

In order to practice the above-described first embodiment, details ofeach component will be described using one exemplary embodiment.

First, a coordinate system that is a precondition for the presentembodiment will be described. A coordinate system that relates to animage data item illustrated in FIG. 3 and that is used in the presentembodiment will now be described. The coordinate system is formed on animage area that has 2048×2048 pixels and that can be captured by thecamera 114, and in the image area, a direction toward the right side inthe horizontal direction as viewed in FIG. 3 is defined as the xdirection, and a direction toward the lower side in the verticaldirection as viewed in FIG. 3 is defined as the y direction. The origin(x=1, y=1) of the coordinate system is the point located at the top leftcorner of the image area. For example, in the image area, thecoordinates of the point located at the top right corner, thecoordinates of the point located at the bottom left corner, and thecoordinates of the point located at the bottom right corner arerespectively (x=2048, y=1), (x=1, y=2048), and (x=2048, y=2048).

FIG. 3 illustrates an image data item obtained by capturing an image ofthe robot hand 113 that is holding the component 112.

The component 112 is an object that forms part of a product and has asize of, for example, 50 mm×50 mm or smaller. The component 112 isplaced on a pallet and is not strictly positioned. The component 112 isto be picked up by the robot hand 113 and transported to a positionwhere a workpiece (not illustrated) is disposed in order to assemble theproduct. When the component 112 is in a state of being held by the robothand 113, the orientation of the component 112 has not yet beendetermined. After an image has been captured as “visual information” bythe apparatus for capturing an image of a moving object according to thepresent invention, the position and orientation of the component 112 ismeasured through image processing, and the position and orientation ofthe component 112 is corrected by the robot hand 113. Then, thecomponent 112 is mounted on the workpiece.

The robot hand 113, which has been installed on a workbench, includes,for example, three fingers and can stably hold the component 112 andtransport the component 112 along a path along which the robot hand 113has been previously instructed to move in a work area that is formed inthe workbench and that has a size of 500 mm×500 mm. The robot hand 113transports the component 112 at a velocity of 2,000 mm/sec. In thepresent embodiment, the robot hand 113 is instructed beforehand to movein the field of view of the camera 114 at a constant velocity of 2,000mm/sec while holding the component 112. In this case, the robot hand 113is instructed to move in such a manner that the component 112 crossesnear the center of the field of view of the camera 114, and that amovement direction of the component 112 is close to the x direction inthe image area, which is captured. In other words, the robot hand 113 isinstructed to move in such a manner that the component 112 appears fromthe left end (x=1) of the image area, moves substantially linearly inthe x direction, and disappears to the right end (x=2048) of the imagearea. Note that a surface of the robot hand 113 that is to be capturedby the camera 114, which will be described below, is painted black, andthe color of the background of the robot hand 113 that is to be capturedby the camera 114 is also black. These are conducted as devices forsuppressing generation of undesirable reflected light. In the presentembodiment, the center (x=1024) of the image in the x direction is atarget capturing position. Obviously, the target capturing position maybe an arbitrary x coordinate or may be an arbitrary coordinate not inthe x direction but in the y direction depending on the environments inwhich image capturing is performed and the like.

The image pickup element 101, the lens 102, and the computing section115 that are included in the camera 114 will now be described.

The image pickup element 101 has a spatial resolution of about 4megapixels (2048×2048 pixels), and each pixel have a bit depth of 8bits. The image pickup element 101 outputs an image by 10 pairs of LVDSsignals. In the case of outputting 4 megapixels, the frame rate is, forexample, 160 fps. An image-capturing-optical system that is acombination of the image pickup element 101 and the lens 102 has acapturing field of view of 100 mm×100 mm and a pixel resolution of about50 μm×50 μm. The image pickup element 101 writes a set value into aregister, which is embedded in the image pickup element 101, throughcommunication using a serial peripheral interface (SPI) or the like, andthen reads an image of about 4 megapixels (2048×2048 pixels). The imagepickup element 101 has the movie recording mode in which the imagepickup element 101 continuously captures images at a fixed frame rate.In each captured frame, while the image pickup element 101 istransmitting a signal, a vertical synchronizing signal is high. In thepresent embodiment, first, a set value that causes the image pickupelement 101 to operate in the movie recording mode is written into theregister, which is embedded in the image pickup element 101, via thecontroller 103 in such a manner as to set the operation mode of theimage pickup element 101 to the movie recording mode. An image that iscaptured by the image pickup element 101 is output to the image inputinterface 104, which will be described later.

The computing section 115 is an electronic circuit board equipped withthe controller 103, the image input interface 104, theimage-feature-amount-calculation unit 106, the memory 107, the imageselection unit 108, and the image output interface 109. The controller103, the image-feature-amount-calculation unit 106, and the imageselection unit 108 are mounted as, for example, computing blocks in anFPGA device, which is to be mounted on the electronic circuit board.Each of the computing blocks is mounted by a combining circuit based ona hardware description using a hardware description language (HDL),which is commonly known, and a macro circuit included in the FPGA.Considering circuit area, manufacturing costs, and performance balance,the image input interface 104 and the image output interface 109 may bemounted in the FPGA, or the computing section 115 may be formed withoutusing the FPGA.

The image input interface 104 uses a commonly known deserializer IC thatconverts LVDS signals, which are input to the image input interface 104from the image pickup element 101, into parallel signals that are easyto use in an electronic circuit. The term “LVDS” refers to “low voltagedifferential signal”. Alternatively, LVDS signals may be input to acommonly known field programmable gate array (FPGA), which is anintegrated circuit whose function can be changed and which is capable ofreceiving LVDS signals, and inverted into parallel signals. As thedeserializer IC, a device to which 10 differential pairs of LVDS signalscan be input is used. Alternatively, a plurality of devices each havinga capacity insufficient to receive 10 differential pairs of LVDS signalsmay be used in parallel as the deserializer IC. Here, signals that areoutput from the image input interface 104 are an 80-bit parallel signal(8 bits×10 TAP), a pixel clock signal, a horizontal synchronizingsignal, and a vertical synchronizing signal, and these signals areoutput to the image-feature-amount-calculation unit 106.

The image-feature-amount-calculation unit 106 of the computing section115 is a computing block that is mounted in the FPGA device mounted onthe electronic circuit board. The image-feature-amount-calculation unit106 detects the position of the component 112 in an image by using animage signal that is formed of the 80-bit parallel signal, the pixelclock signal, the horizontal synchronizing signal, and the verticalsynchronizing signal, which have been input to theimage-feature-amount-calculation unit 106 from the image input interface104. In addition, the image-feature-amount-calculation unit 106 detectsa total value of the luminance of the image as the image feature valuecorrelated with illumination conditions. In the present embodiment,since the color of the robot hand 113 and the color of the background ofthe robot hand 113 when an image of the robot hand 113 is captured areblack, image signals that are input to theimage-feature-amount-calculation unit 106 indicate images such as thoseillustrated in FIG. 4. First, binarization processing is performed on aparallel image signal that has been input per 8 bits, which is onepixel, in such a manner as to acquire a binary image. The binarizationprocessing is performed by setting the pixel value to HIGH (1) when apixel value is greater than a predetermined threshold (e.g., 128) andLOW (0) when the pixel value is not greater than the threshold. Theimages illustrated in FIG. 5 are examples of such a binary image.However, in the present embodiment, the binarization processing isperformed by pipeline processing at a pixel level as described later,and thus, it should be noted that a group of binary images such as thoseillustrated in FIG. 5 will not be stored or output. Next, the centroidof the binary image that denotes the positional information of thecomponent 112 is calculated.

A method for calculating the centroid of a binary image having pixelswhose values are 0 or 1 will now be described. The centroid of an imagegenerally denotes the central coordinates of a mass distribution whenluminance value is considered as mass, and in a binary image, thecentroid of an image is the central coordinates of a plurality of pixelseach having a luminance value of 1. In order to calculate the centroidof an image, the zeroth moment of the image and the first moment of theimage are used. In general, such a moment of an image also denotes agravitational moment when luminance value is considered as mass. Thezeroth moment of a binary image denotes the total number of pixels eachhaving a luminance value of 1, and the first moment of a binary imagedenotes the sum of positional coordinate values of pixels each having aluminance value of 1. In the present embodiment, the first moment of animage that is calculated with respect to the horizontal direction isreferred to as the horizontal first moment of an image, and the firstmoment of an image that is calculated with respect to the verticaldirection is referred to as the vertical first moment of an image. Thehorizontal coordinates of the centroid of an image can be calculated bymultiplying the horizontal first moment of the image by the reciprocalof the zeroth moment of the image. The vertical coordinates of thecentroid of an image can be calculated by multiplying the vertical firstmoment of the image by the reciprocal of the zeroth moment of the image.

Based on the above, the zeroth moment of the image, the horizontal firstmoment of the image, and the vertical first moment of the image arecalculated with respect to the binary signal, which has been obtained.In the image-feature-amount-calculation unit 106, the followingresisters are included in the computing block of the FPGA. That is tosay, the image-feature-amount-calculation unit 106 includes a horizontalcoordinate register, which is to be incremented in synchronization witha pixel clock and reset in synchronization with a horizontalsynchronizing signal, and a vertical coordinate register, which is to beincremented in synchronization with a horizontal synchronizing signaland reset in synchronization with a vertical synchronizing signal. Inaddition, the image-feature-amount-calculation unit 106 includes azeroth moment register, which holds the integral value of the zerothmoment of an image, a horizontal first moment resister, which holds theintegral value of the horizontal first moment of an image, and avertical first moment resister, which holds the integral value of thevertical first moment of an image. Each of the resisters holds zero asan initial value. First, when a one-bit binary image signal is input, aone-bit value is added to the value stored in the zeroth momentregister. At the same time, a calculation (bit value×(value inhorizontal coordinate register)) is performed, and the result is addedto the value stored in the horizontal first moment resister. At the sametime, a calculation (bit value×(value in vertical coordinate register))is performed and the result is added to the value stored in the verticalfirst moment resister. The above calculations are repeated for thenumber of times corresponding to all the pixels (2048×2048 pixels) insynchronization with a pixel clock. As a result, in the entire image,the zeroth moment of the image, the horizontal first moment of theimage, and the vertical first moment of the image are respectivelystored in the zeroth moment register, the horizontal first momentresister, and the vertical first moment resister.

Next, the centroid of the image is calculated from the zeroth moment ofthe image, the horizontal first moment of the image, and the verticalfirst moment of the image, which have been calculated. The horizontalcoordinates of the centroid of the image is calculated by a hardwarecomputation using a formula (horizontal first moment resistervalue/zeroth moment register value). The vertical coordinates of thecentroid of the image is calculated by a hardware computation using aformula (vertical first moment resister value/zeroth moment registervalue). The horizontal coordinates of the centroid of the image and thevertical coordinates of the centroid of the image, which have beencalculated as described above, form the positional information of thecomponent 112.

Regarding a series of image data items such as those illustrated in FIG.4 that are to be input to the image-feature-amount-calculation unit 106of the computing section 115, a total value of the luminance of each ofthe image data items is calculated. In theimage-feature-amount-calculation unit 106, a luminance-total-valueresister that holds the integral value of a total value of the luminanceof an image in synchronization with a pixel clock is included in thecomputing block of the FPGA. The luminance-total-value resister holdszero as an initial value. First, when an 8-bit image signal is input, an8-bit value is added to the value stored in the luminance-total-valueresister. This calculation is repeated for the number of timescorresponding to all the pixels (2048×2048 pixels) in synchronizationwith a pixel clock, and as a result, in the entire image, the totalvalue of the luminance of the image is stored in theluminance-total-value resister.

The positional information of the component 112 and the total value ofthe luminance of the image, which have been calculated through the abovecomputations, are output with the corresponding image data items fromthe image-feature-amount-calculation unit 106 and input to the imageselection unit 108. Here, in the case where the component 112 is notpresent in the capturing field of view of the camera 114, “0” is outputas positional information. Note that, among the above-describedcomputations, the binarization processing performed on each pixel andaccumulative calculations for the zeroth moment of the image, thehorizontal first moment of the image, and the vertical first moment ofthe image are performed by pipeline processing. In other words, forexample, instead of waiting until the binarization processing performedon all the pixels is completed, the accumulative calculation of a firstpixel is performed while the binarization processing is performed on asecond pixel, and the accumulative calculation of the second pixel isperformed while the binarization processing is performed on a thirdpixel.

In the present embodiment, the method in which the centroid of an imageis calculated after performing binarization processing on the image andcalculating the zeroth moment of the image, the horizontal first momentof the image, and the vertical first moment of the image has beendescribed as a method for detecting the position of the component 112 inan image. However, a commonly known method for detecting an object thatis different from the above method and that is based on the fact that abackground is black may be used. For example, a template image of thecomponent 112, the template image having a resolution corresponding toan image captured by the camera 114, may be stored beforehand in theFPGA, a processing circuit that performs a commonly known templatematching may be mounted in the FPGA, and the position of the component112 may be detected. In the case where, despite the absence of thecomponent 112 in the capturing field of view of the camera 114, thecomponent 112 is detected due to noise mixed in an image, filteringprocessing in which the value of output coordinates is set to “0” whilethe value of the zeroth moment of the image is set to be a threshold maybe performed.

In addition, in the description of the present embodiment, although atotal value of the luminance of an image has been used as the imagefeature value correlated with illumination conditions, a different imagefeature value, such as those described above, may be used as long as ithas been previously confirmed that the image feature value is correlatedwith illumination conditions.

The memory 107 is a random access memory (RAM) mounted on the electroniccircuit board and is formed of a plurality of 256-Kbyte synchronousdynamic RAMs (SDRAMs). The plurality of SDRAMs includes, for example, 10SDRAMs. The bit width of each of the SDRAMs is 8 bits. Each of theSDRAMs specifies a row address and a column address and is capable ofperforming reading and writing in synchronization with a synchronizingsignal.

The image selection unit 108 is a computing block that is mounted in theFPGA device mounted on the electronic circuit board. The image selectionunit 108 acquires the positional information of the component 112 thatis formed of the horizontal coordinates of the centroid of an image andthe vertical coordinates of the centroid of the image from theimage-feature-amount-calculation unit 106. A total value of theluminance of the image and an image signal, which is formed of an 80-bitparallel signal, a pixel clock signal, a horizontal synchronizingsignal, and a vertical synchronizing signal, are input to the imageselection unit 108. In addition, the reference image feature values areinput to the image selection unit 108 from the controller 103. The imageselection unit 108 includes a memory interface that is used foraccessing the memory 107. The memory interface is provided with 10computing blocks that are connected in parallel in such a manner as tocorrespond to the 10 SDRAMs. When the vertical synchronizing signalbecomes HIGH, the memory interface starts memory access and supplies apixel clock signal as a synchronizing signal used in the memory accessto the SDRAMs. In addition, the memory interface increments a rowaddress in synchronization with the pixel clock signal and increments acolumn address in synchronization with a horizontal synchronizing signalin such a manner as to set an address used for accessing the SDRAMs.When the vertical synchronizing signal becomes LOW, the memory interfaceterminates the memory access. An image data item of one frame that hasimage feature values that most closely match the reference image featurevalues and the image feature values are stored in the memory 107. In aninitial state, “0” is stored as an image data item and image featurevalues in the memory 107.

During the period when the positional information of the component 112,which has been input, is “0”, that is, during the period when thecomponent 112 has not yet reached the field of view of the camera 114,the image selection unit 108 does not perform an operation and waits foran image data item and image feature values that are to be input next.Then, when the positional information of the component 112 has changedfrom “0” to a value other than “0”, the image selection unit 108compares the reference image feature values stored in the memory 107 andthe image feature values that have been input. After that, the imageselection unit 108 determines whether the degree of matching between theinput image feature values and the reference image feature values, whichhave been input from the controller 103, is higher than the degree ofmatching between the image feature values of an image stored in thememory 107 and the reference image feature values, which have been inputfrom the controller 103. In the case where the degree of matchingbetween the input image feature values and the reference image featurevalues input from the controller 103 is higher than the degree ofmatching between the image feature values of the image stored in thememory 107 and the reference image feature values input from thecontroller 103, the image selection unit 108 updates the contents of thememory 107 to the input image data item and the corresponding imagefeature values, or otherwise, the image selection unit 108 does notupdate the contents of the memory 107 and discards the input image dataitem and the corresponding image feature values.

Finally, when the positional information of the component 112 is changedto “0” from a value other than “0”, the image selection unit 108 readsthe image data item stored in the memory 107 at the time and outputs theimage data item to the image output interface 109.

A method for determining whether the degree of matching between theimage feature values, which have been input, and the reference imagefeature values is higher than the degree of matching between the imagefeature values stored in the memory 107 and the reference image featurevalues will now be described. First, it is determined that the entiretyof the component 112 is located in the field of view of the camera 114.Referring to the positional information of the component 112 included inthe input image feature values, in the case where the horizontalcoordinates of the centroid of the image is 513 or greater and 1,536 orsmaller, and where the vertical coordinates of the centroid of the imageis 513 or greater and 1,536 or smaller, the rank of the image featurevalues is determined as A, or otherwise determined as B.

Next, the total value of the luminance of the image is evaluated. Theabsolute value of the difference between the total value of theluminance of the image, which is included in the input image featurevalues, and the reference image feature values is calculated.

Finally, it is determined which one of the degree of matching betweenthe image feature values stored in the memory 107 and the referenceimage feature values and the degree of matching between the input imagefeature values and the reference image feature values is higher, and theinput image feature values are rated as “Excellent” or “Poor”. The rankof the image feature values stored in the memory 107 is determined asone of A, B, or “0” by a method similar to the method for determiningthe rank of the input image feature values. In addition, the absolutevalue of the difference between the total value of the luminance of theimage and the reference image feature values is calculated. In the casewhere the image feature values stored in the memory 107 are “0”, theinput image feature values are rated as “Excellent”. In the case wherethe rank of the image feature values stored in the memory 107 is B, ifthe rank of the input image feature values is A, the input image featurevalues are rated as “Excellent”. In the case where the rank of the inputthe image feature values is B, if the absolute value of the differencebetween the input image feature values and the reference image featurevalues is not greater than the image feature values stored in the memory107, the input image feature values are rated as “Excellent”, and if theabsolute value is greater than the image feature values stored in thememory 107, the input image feature values are rated as “Poor”. In thecase where the rank of the image feature values stored in the memory 107is A, if the rank of the input image feature values is B, the inputimage feature values are rated as “Poor”. In the case where the rank ofthe input image feature values is A, if the absolute value of thedifference between the input image feature values and the referenceimage feature values is not greater than the image feature values storedin the memory 107, the input image feature values are rated as“Excellent”, and if the absolute value is greater than the image featurevalues stored in the memory 107, the input image feature values arerated as “Poor”. In the case where the input image feature values arerated as “Excellent” through the above determination processes, thedegree of matching between the input image feature values and thereference image feature values is considered as high, and the contentsof the memory 107 are updated to the image signal and the image featurevalues, which have been input. If the input image feature values arerated as “Poor”, the contents of the memory 107 will not be updated, andthe input image data item and the corresponding image feature values arediscarded.

In the present embodiment, the process for rating the image featurevalues stored in the memory 107 and the calculation for the absolutevalue of the difference between the total value of the luminance of theimage and the reference image feature values are performed in accordancewith the timing of performing the calculation for the input imagefeature values. However, the results of the rating process that isperformed when the image feature values are stored in the memory 107 andthe results of the calculation for the absolute value of the differencebetween the total value of the luminance of the image and the referenceimage feature values may be stored and reused.

The image output interface 109 uses a serializer IC that converts the80-bit parallel signal, the pixel clock signal, the horizontalsynchronizing signal, and the vertical synchronizing signal, which havebeen input from the image selection unit 108, into LVDS image signalsconforming to Camera Link or the like. Alternatively, using an FPGAdevice that is capable of outputting LVDS signals may be used in such amanner as to convert parallel signals into serial signals in the FPGA.The LVDS signals output from the image output interface 109 are receivedby an external camera link grabber board or the like, and imageprocessing is performed on the LVDS signals by a processor or the like.

The controller 103 is a computing block that is to be mounted in theFPGA device mounted on the electronic circuit board. The controller 103issues a continuous-image-capturing-setting command to the image pickupelement 101 via an SPI interface in such a manner as to cause the imagepickup element 101 to have the movie recording mode. In addition, thecontroller 103 stores the reference image feature values that are inputfrom the outside beforehand into an internal resister and outputs thereference image feature values to the image selection unit 108. Notethat the reference image feature values are the positional informationof the component 112, which is formed of the horizontal coordinates ofthe centroid of an image and the vertical coordinates of an image, andthe total value of the luminance of an image that are calculated from animage data item, which has been used when the image processing algorithmhas been previously determined, by a method that is the same as theabove-described calculation method.

According to the present embodiment, an image that is similar to animage data item, which has been used when the image processing algorithmhas been previously determined, can be captured. Thus, among a pluralityof captured images of the component 112 that moves in the vicinity ofthe target capturing position, only one image that is the best image inwhich image processing is most likely to be properly completed can beoutput. Therefore, even if a frame in which ambient light of, forexample, a fluorescent lamp and the like is reflected is produced, oreven if a frame in which a shadow is reflected is produced due to theamount of displacement with respect to the target capturing position anda change in illumination conditions as a result of swing and tilt, theposition and orientation of the component 112 can be correctlyrecognized.

Second Embodiment

In the above-described first embodiment, the colors of the robot hand113 and the background of the robot hand 113 in an image captured by thecamera 114 are black in such a manner as to provide advantages incapturing an image of the component 112 and subsequent image processing.However, in an actual manufacturing site, such as a manufacturingfactory, there is a probability that other component to be assembled maybe present in the background of an image to be captured. Thus, when animage is captured the color of the background of the image is not alwaysset to black. Even in the case where the entire color of the backgroundis set to black, there is a possibility that the background will bebrightened by reflection of ambient light and the like, and thus, thereis a problem in that there is a possibility that the color of thebackground of an image to be captured may not be set to black.

Accordingly, as a second embodiment, an apparatus for capturing an imageof a moving object that captures an image of a component that is movingunder illumination conditions similar to those under which an image thathas been previously assumed is captured even if the background of theimage to be captured has brightness and that performs a backgroundremoval processing will be described.

FIG. 6 illustrates the configuration of the apparatus for capturing animage of a moving object according to the second embodiment of thepresent invention. The apparatus for capturing an image of a movingobject according to the second embodiment of the present inventionincludes a camera 114, which includes an image pickup element 101, alens 102, and a computing section 115, a component 112, and a robot hand113 that is used for holding a component 112. The computing section 115includes a controller 103, an image input interface 104, animage-feature-value-calculation unit 106, a memory 107, a memory 110, animage selection unit 108, and an image output interface 109. Since theflow of processes when the apparatus for capturing an image of a movingobject captures an image of the component 112, which is moving whilebeing held by the robot hand 113, is similar to that in the firstembodiment, the description of the flow of processes will be omitted,and the robot hand 113 and the image-feature-amount-calculation unit 106will be described.

First, the robot hand 113 will be described. The robot hand 113includes, for example, three fingers and can stably hold the component112 and transport the component 112 along a path along which the robothand 113 has been previously instructed to move in a work area that hasa size of 500 mm×500 mm. The robot hand 113 is capable of transportingthe component 112 at a velocity of up to 2,000 mm/sec. Also in thepresent embodiment, the robot hand 113 moves in the field of view of thecamera 114, which will be described later, at a constant velocity of2,000 mm/sec while holding the component 112. In this case, thecomponent 112 crosses near the center of the field of view of the camera114, and a movement direction of the component 112 is close to the xdirection in an image that is captured. In other words, the robot hand113 is instructed to move in such a manner that the component 112appears from the left end (x=1) of an image area, moves substantiallylinearly in the x direction, and disappears to the right end (x=2048) ofthe image area. Note that optical characteristics, such as color andbrightness, of the background of the robot hand 113 that is to becaptured by the camera 114 do not change over time. In the presentembodiment, the center (x=1024) of the image in the x direction is atarget capturing position. Obviously, the target capturing position maybe an arbitrary x coordinate or may be an arbitrary coordinate not inthe x direction but in the y direction depending on lightingenvironments and the like.

Next, the memory 110 will be described. The memory 110 is a RAM mountedon an electronic circuit board and is formed of a plurality of 256-KbyteSDRAMs. The plurality of SDRAMs includes, for example, 10 SDRAMs. Thebit width of each of the SDRAMs is 8 bits. Each of the SDRAMs specifiesa row address and a column address and is capable of performing readingand writing in synchronization with a synchronizing signal.

Next, the image-feature-amount-calculation unit 106 will be described.The image-feature-amount-calculation unit 106 is a computing block thatis mounted in an FPGA device mounted on the electronic circuit board.The image-feature-amount-calculation unit 106 detects the position ofthe component 112 in an image by using an image signal that is formed ofan 80-bit parallel signal, a pixel clock signal, a horizontalsynchronizing signal, and a vertical synchronizing signal, which havebeen input to the image-feature-amount-calculation unit 106 from theimage input interface 104. In addition, theimage-feature-amount-calculation unit 106 detects a total value of theluminance of an image as the image feature value correlated withillumination conditions. The image-feature-value-calculation unit 106includes a memory interface that is used for accessing the memory 107.The memory interface is provided with 10 computing blocks that areconnected in parallel in such a manner as to correspond to the 10SDRAMs. When the vertical synchronizing signal becomes HIGH, the memoryinterface starts memory access and supplies a pixel clock signal as asynchronizing signal used in the memory access to the SDRAMs. Inaddition, the memory interface increments a row address insynchronization with the pixel clock signal and increments a columnaddress in synchronization with a horizontal synchronizing signal insuch a manner as to set an address used for accessing the SDRAMs. Whenthe vertical synchronizing signal becomes LOW, the memory interfaceterminates the memory access. An image signal of a frame immediatelybefore the last frame is stored in the memory 110.

In the present embodiment, image signals that are input to theimage-feature-amount-calculation unit 106 indicate images such as thoseillustrated in FIG. 7. First, a total value of the luminance of an imagewith relevant to a parallel image signal that has been input iscalculated. In the image-feature-amount-calculation unit 106, aluminance-total-value resister that holds the integral value of a totalvalue of the luminance of an image in synchronization with a pixel clockis included in the computing block of the FPGA. Theluminance-total-value resister holds zero as an initial value. First,when an 8-bit image signal is input, an 8-bit value is added to thevalue stored in the luminance-total-value resister. This calculation isrepeated for the number of times corresponding to all the pixels(2048×2048 pixels) in synchronization with a pixel clock, and as aresult, in the entire image, the total value of the luminance of theimage is stored in the luminance-total-value resister.

A process for calculating the positional information of the component112 that is to be performed along with the above processing will now bedescribed.

First, the difference value between the last image and an image that isone frame before the last image and that is read from the memory 110 viathe memory interface in synchronization with a parallel image signal, apixel clock signal, a horizontal synchronizing signal, and a verticalsynchronizing signal, that are input, is calculated on a pixel-by-pixelbasis. Next, binarization processing is performed on the differencevalue of each pixel. The binarization processing is performed by settingthe pixel value to HIGH (1) when the pixel value is greater than apredetermined threshold (e.g., 128) and to LOW (0) when the pixel valueis not greater than the threshold. Examples of images that have beenbinarized are illustrated in FIG. 8. However, in the present embodiment,the binarization processing is performed by pipeline processing at apixel level as described later, and thus, it should be noted that agroup of binary images such as those illustrated in FIG. 8 will not bestored or output.

Next, the zeroth moment of the image, the horizontal first moment of theimage, and the vertical first moment of the image are calculated withrespect to the binary signal, which has been obtained. In theimage-feature-amount-calculation unit 106, the following resisters areincluded in the computing block of the FPGA. That is to say, theimage-feature-amount-calculation unit 106 includes a horizontalcoordinate register, which is to be incremented in synchronization witha pixel clock and reset in synchronization with a horizontalsynchronizing signal, and a vertical coordinate register, which is to beincremented in synchronization with a horizontal synchronizing signaland reset in synchronization with a vertical synchronizing signal. Inaddition, the image-feature-amount-calculation unit 106 includes azeroth moment register, which holds the integral value of the zerothmoment of an image, a horizontal first moment resister, which holds theintegral value of the horizontal first moment of an image, and avertical first moment resister, which holds the integral value of thevertical first moment of an image. Each of the resisters holds zero asan initial value. First, when a one-bit binary image signal is input, aone-bit value is added to the value stored in the zeroth momentregister. At the same time, a calculation (bit value×(value inhorizontal coordinate register)) is performed, and the result is addedto the value stored in the horizontal first moment resister. At the sametime, a calculation (bit value×(value in vertical coordinate register))is performed and the result is added to the value stored in the verticalfirst moment resister. The above calculations are repeated for thenumber of times corresponding to all the pixels (2048×2048 pixels) insynchronization with a pixel clock. As a result, in the entire image,the zeroth moment of the image, the horizontal first moment of theimage, and the vertical first moment of the image are respectivelystored in the zeroth moment register, the horizontal first momentresister, and the vertical first moment resister.

Next, the centroid of the image is calculated from the zeroth moment ofthe image, the horizontal first moment of the image, and the verticalfirst moment of the image, which have been calculated. The x coordinateof the centroid of the image is calculated by a hardware computationusing a formula (horizontal first moment resister value/zeroth momentregister value). The y coordinate of the centroid of the image iscalculated by a hardware computation using a formula (vertical firstmoment resister value/zeroth moment register value). The positionalinformation of the component 112 and the total value of the luminance ofthe image, which are calculated by the above computations, are outputfrom the image-feature-amount-calculation unit 106 and input to theimage selection unit 108. Here, in the case where the component 112 isnot present in the capturing field of view of the camera 114, “0” isoutput as positional information. Note that, among the above-describedcomputations, the binarization processing performed on each pixel andaccumulative calculations for the zeroth moment of the image, thehorizontal first moment of the image, and the vertical first moment ofthe image are performed by pipeline processing. In other words, forexample, instead of waiting until the binarization processing performedon all the pixels is completed, the accumulative calculation of a firstpixel is performed while the binarization processing is performed on asecond pixel, and the accumulative calculation of the second pixel isperformed while the binarization processing is performed on a thirdpixel. Finally, the contents of the memory 110 is overwritten with theparallel image signal, which has been input, via the memory interface bythe pixel clock signal, the horizontal synchronizing signal, and thevertical synchronizing signal, and an image of one frame is stored intothe 10 SDRAMs. The image that is stored will be used as an image that isone frame before the next frame.

Although, in the present embodiment, the method for removing thebackground of an image of the component 112 that is captured by aninter-frame difference method has been described, a commonly knownbackground removal processing that is different from the method may beused. For example, a background image in which the component 112 and therobot hand 113 do not appear may be stored in the memory 110 beforehandby using the memory 110 as a non-volatile memory, and the background maybe removed by calculating the difference between the background imageand an image to be captured, or other algorithms for removing thebackground may be used. In the case where, despite the absence of thecomponent 112 in the capturing field of view of the camera 114, thecomponent 112 is detected due to noise mixed in an image, filteringprocessing in which the value of output coordinates is set to “0” whilethe value of the zeroth moment of the image is set to be a threshold maybe performed. In addition, in the description of the present embodiment,although a total value of the luminance of an image has been used as theimage feature value correlated with illumination conditions, a differentimage feature value, such as those described above, may be used as longas it has been previously confirmed that the image feature value iscorrelated with illumination conditions.

According to the present embodiment, image capturing can be efficientlyperformed even if the background of an image of the robot hand 113 thatis to be captured by the camera 114 is not black and has brightness. Inother words, among a plurality of captured images of the component 112that moves in the vicinity of the target capturing position, one imagethat is the best image in which image processing is most likely to beproperly completed can be output. Thus, in a situation where it isdifficult to set the color of the background of an image to black, theposition and orientation of the component 112 can be correctlyrecognized.

By performing a method for producing an item that is to be performed bya robot apparatus that includes a camera, which performs the imagecapturing method according to the present invention, and a robot hand,an image of an object held by the robot hand can be effectively capturedwhile the object is moving, and thus, an production activity can beeffectively performed.

The present invention can be used in an assembling robot system.

According to the present invention, an image to be output is selectedwhile the position of a component and also an image feature value thatis correlated with illumination conditions under which images have beencaptured function as references. Thus, in the case where a componentthat is moving has been inclined due to vibration and the like generatedduring transportation of the component, an image that has an imagefeature value equivalent to a sample image can be selected and output.Therefore, an image can be captured under illumination conditionssimilar to those under which an image that has been assumed is captured,and thus, the present invention has an advantageous effect in that theposition and orientation of a component that is moving can be correctlyrecognized.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

1. An image processing method comprising: acquiring at least two imagedata items obtained by capturing images of a same object within a fieldof view of an image-capturing-optical system; calculating an imagefeature value for each of the two image data items, the image featurevalue correlated with a difference in brightness of at least a part ofthe object within the field of view of the image-capturing-opticalsystem; and performing processes different from one another for the twoimage data items, respectively, depending on the image feature values ofthe respective two image data items.
 2. The image processing methodaccording to claim 1, wherein the object moves between a time when animage of the object is captured to acquire one of the two image dataitems and a time when an image of the object is captured to acquireanother one of the two image data items.
 3. The image processing methodaccording to claim 1, wherein one of the processes performed for the twoimage data items includes a process for discarding one of the two imagedata items.
 4. The image processing method according to claim 1, whereinthe processes for the respective two image data items are determinedbased on a degree of matching between each of the image feature valuesof the two image data items and a reference value that has been obtainedbefore acquiring the two image data items.
 5. The image processingmethod according to claim 1, wherein the processes are performed for thetwo image data items, respectively, after acquiring the two image dataitems.
 6. The image processing method according to claim 1, wherein atleast one of a position of a light source, an irradiation intensity, anirradiation angle, and a light-emitting wavelength of an illuminationunit configured to illuminate the object differs between a time when animage of the object is captured to acquire one of the two image dataitems and a time when an image of the object is captured to acquireanother one of the two image data items.
 7. A non-transitory storagemedium storing a program that, when executed by a computer, executes theimage processing method according to claim
 1. 8. A system comprising: acomputing unit configured to acquire at least two image data items ofimages captured by an image-capturing-optical system, wherein thecomputing unit calculates an image feature value for each of the twoimage data items, the image feature value correlated with a differencein brightness of at least a part of an object within a field of view ofthe image-capturing-optical system, and wherein, after the computingunit acquires the two image data items, the computing unit performsprocesses different from one another for the two image data items,respectively, depending on the image feature values of the respectivetwo image data items.
 9. The system according to claim 8, wherein one ofthe processes performed for the two image data items includes a processfor discarding one of the two image data items.
 10. The system accordingto claim 9, wherein the other of the processes performed for the twoimage data items includes a process for saving the other of the twoimage data items and saving the image feature value of the other of thetwo image data items.
 11. The system according to claim 8, wherein theother of the processes performed for the two image data items includes aprocess for generating a video signal based on the other of the twoimage data items.
 12. The system according to claim 8, wherein the imagefeature value is a value based on a sum of values indicating brightnessof each of a plurality of pixels in the image data item.
 13. The systemaccording to claim 8, wherein the computing unit determines theprocesses for the respective two image data items based on the imagefeature value of one of the two image data items and the image featurevalue of the other of the two image data items.
 14. The system accordingto claim 8, wherein the computing unit determines the processes for therespective two image data items based on a degree of matching betweeneach of the image feature values of the two image data items and areference value that has been obtained before acquiring the two imagedata items.
 15. The system according to claim 14, wherein an image dataitem, of the two image data items, having a lower degree of matching isdiscarded.
 16. The system according to claim 8, wherein the computingunit performs a process for removing information corresponding tobrightness of a background of the object in each of the two image dataitems, before performing the processes different from one another forthe two image data items, respectively.
 17. The system according toclaim 8, further comprising an illumination unit configured toilluminate the object, wherein the computing unit controls theillumination unit in such a manner that at least one of a position of alight source, an irradiation intensity, an irradiation angle, and alight-emitting wavelength of the illumination unit differs between atime when an image of the object is captured to acquire one of the twoimage data items and a time when an image of the object is captured toacquire another one of the two image data items.
 18. The systemaccording to claim 8, further comprising a moving unit configured tomove the object.
 19. The system according to claim 8, wherein a positionand orientation of the object is recognized using any one of the twoimage data items.
 20. A system comprising: an image-capturing-opticalsystem; an illumination unit configured to illuminate an object within afield of view of the image-capturing-optical system; and a computingunit configured to acquire image data of an image captured by animage-capturing-optical system, wherein the illumination unit iscontrolled in such a manner that a brightness of at least a part of theobject within the field of view of the image-capturing-optical systemdiffers between a time when an image of the object is captured by theimage-capturing-optical system to acquire one of the two image dataitems and a time when an image of the object is captured to acquireanother one of the two image data items, and wherein the computing unitperforms a process for each of the two image data items based on thedifference in the brightness of at least a part of the object within thefield of view of the image-capturing-optical system.
 21. The systemaccording to claim 20, wherein the computing unit performs processesdifferent from one another for the two image data items, respectively.22. A production method for producing an article using the imageprocessing method according to claim 1, wherein the object is at least apart of the article.
 23. A production method for producing an articleusing the system according to claim 8, wherein the object is at least apart of the article.
 24. A production method for producing an articleusing the system according to claim 20, wherein the object is at least apart of the article.
 25. A camera comprising: an image-capturing-opticalsystem; and a computing unit configured to acquire image data of animage captured by an image-capturing-optical system, wherein, when thecomputing unit acquires two image data items from capturing images of asame object within a field of view of an image-capturing-optical system,and in a case where a brightness of at least a part of the object withinthe field of view of the image-capturing-optical system differs betweena time when an image of the object is captured to acquire one of the twoimage data items and a time when an image of the object is captured toacquire another one of the two image data items, the computing unitperforms, after capturing images of the object to acquire the two imagedata items, a process for discarding one of the two image data items anda process for outputting a video signal based on the other of the twoimage data items to outside the camera.